Use weblog metadata to declare weblog's category, and connect to scenario matrix builder#7286
Use weblog metadata to declare weblog's category, and connect to scenario matrix builder#7286cbeauchesne wants to merge 10 commits into
Conversation
|
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: e937426 | Docs | Datadog PR Page | Give us feedback! |
cbeauchesne
changed the title
robertomonteromiguel
left a comment
robertomonteromiguel
left a comment
There was a problem hiding this comment.
Add a test_the_test check that verifies every scenario name listed in supported_scenarios/excluded_scenarios in the YAML files is a valid scenario name.
| other_weblog_containers=(PostgresContainer,), | ||
| scenario_groups=[scenario_groups.essentials, scenario_groups.telemetry], | ||
| weblog_categories=[WeblogCategory.dd_trace], | ||
| doc="Default scenario, spawn tracer, the Postgres databases and agent, and run most of exisiting tests", |
There was a problem hiding this comment.
I'm sorry, I don't see it 🤔
cbeauchesne
requested review from
amarziali,
asopkin,
daniel-romano-DD,
manuel-alvarez-alvarez and
sabrenner
and removed request for
a team
Done in e937426 |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: da0b94e39f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| appsec_enabled=self.appsec_enabled, | ||
| use_proxy_for_agent=False, | ||
| use_proxy_for_weblog=False, | ||
| weblog_categories=[WeblogCategory.dd_trace], |
There was a problem hiding this comment.
Because PerformanceScenario still inherits the default github_workflow="endtoend", adding the dd_trace category here makes workflow_data._filter_scenarios treat every regular dd-trace weblog as supporting PERFORMANCES whenever that scenario is selected for the workflow matrix (for example with --scenarios PERFORMANCES or the exotics group). The old _is_supported explicitly returned false for PERFORMANCES, and the new compatibility test even skips it as a special case that is not inside the workflow, so this change can unexpectedly schedule a large end-to-end CI matrix for a scenario that should stay out of that builder.
Useful? React with 👍 / 👎.
| profiling = ProfilingScenario("PROFILING") | ||
|
|
||
| trace_stats_computation = EndToEndScenario( | ||
| trace_stats_computation = DdTraceEndToEndScenario( |
There was a problem hiding this comment.
Changing these scenarios from exact EndToEndScenario instances to DdTraceEndToEndScenario subclasses means tests/test_the_test/test_minimal_number_of_scenarios.py no longer inspects them, because it filters with type(scenario) is EndToEndScenario. In the TEST_THE_TEST scenario this leaves nearly all regular end-to-end scenarios out of the merge/duplicate-configuration check, so newly duplicate scenario definitions can land unnoticed unless the test is updated to include the new subclasses.
Useful? React with 👍 / 👎.
4 participants
Motivation
Knowning which scenario runs on which weblog is quite complex. So far, it was done by one of the crappiest piece of code I've ever wrote : hardcoded names with organic rules that are added on top of each over threw times.
SSI and libinjection also has it own recipes, with a cleaner approach, where dat is stored in json, and logic in python.
As we'll create an official workflow in gitlab, we need a unified (and clean) way of doing so.
This PR propose an unified way of tacking this, and implement it to end-to-end scenario, SSI will come later
Changes
categoriesfield on weblog metadata data. This category describe what library is shipped inside the weblog :dd_trace: the basic dd-trace HTTP instrumentationdd_trace_graphql: a dd-trace GraphQL instrumentationdd_trace_lambda: a dd-trace lambda instrumentationdd_trace_frameworks: a dd-trace framework instrumentation (mostly AI)open_telemetry: an Open Telemetry instrumentationweblog_categoriesproperty onScenarioThe logic is quite simple : a scenario supports a given weblog if both of them has the same weblog category.
On top of that, weblog metadata has two extra field :
supported_scenariosandexcluded_scenarios, for any fine tune inclusion/exclusion.Few notes :
ymltojsonto store data, as it's easier to read for human (as long as we don't use crazy yml syntax)Workflow
🚀 Once your PR is reviewed and the CI green, you can merge it!
🛟 #apm-shared-testing 🛟
Reviewer checklist
tests/ormanifests/is modified ? I have the approval from R&P teambuild-XXX-imagelabel is present